Comparing of Traditional Assessment with Dynamic Testing in a Tutoring System

It is presumed that if you have a limited amount of time for assessing student knowledge, that you should give a test that does not waste time giving students feedback. Feng and Heffernan (2010) compared two such conditions and found the counter intuitive results that the condition that let them get feedback was actually superior (not statistically reliable) than the “test” condition, in which student did about double the number of problems. The gain in assessment value for the tutor condition came in the form of the addition measure like number of hints a student needed, and how many attempts they had to make before they got the problem correct. Trivedi, Pardos, and Heffernan (2011) analyzed the same data set and found a more powerful way of using the information available in the dynamic testing condition (i.e., the tutor condition) by applying clustering techniques. The found a reliable improvement over the work in Feng and Heffernan (2010) that made the advantage of tutoring appear a decisively better choice if you want to assess student (and obviously a better choice if you care about student learning from feedback). The weakness in Feng & Heffernan (2010) was in its being a simulated condition. In this study, we explain that weakness, and run a new randomized control trial to see if the main effects would replicate. In this paper, we attempted to combine the two studies and eliminate the “weakness by running a randomized controlled experiment in a tutoring system with participants from two different grades, 7 and 8. Our results suggest, that unlike our previous results 1) there is no reliable main effect across all students; 2) the dynamic testing condition works better with7th graders than with 8 graders. We found that 7 graders and 8 graders behaved differently while working within the system. 8 graders asking for significantly more hints than 7 graders when working on the same set of assignments, which appeared to be related to “gaming”.